The  Role  of  the  Revised  IEEE  Standard  Dictionary 
of  Measures  of  the  Software  Aspects  of 
Dependability  in  Software  Acquisition 

•  Dr.  Norman  F.  Schneidewind 
•  Naval  Postgraduate  School 
•  nschneid@nps.navy.mil 


1 


Report  Documentation  Page 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

JAN  2003 


2.  REPORT  TYPE 


3.  DATES  COVERED 

00-00-2003  to  00-00-2003 


4.  TITLE  AND  SUBTITLE 

The  Role  of  the  Revised  IEEE  Standard  Dictionary  of  Measures  if  the 
Software  Aspects  of  Dependability  in  Software  Acquisition 

6.  AUTHOR(S) 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Naval  Postgraduate  School, 1  University  Circle, Monterey, CA, 93943 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 


5a.  CONTRACT  NUMBER 


5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

5d.  PROIECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 


13.  SUPPLEMENTARY  NOTES 


14.  ABSTRACT 


15.  SUBIECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF: 


a.  REPORT 

unclassified 


b.  ABSTRACT 

unclassified 


c.  THIS  PAGE 

unclassified 


17.  LIMITATION  OF 

18.  NUMBER 

ABSTRACT 

OF  PAGES 

Same  as 

21 

Report  (SAR) 

19a.  NAME  OF 
RESPONSIBLE  PERSON 


Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


Outline 


•  Introduction 

•  Rationale  for  Revision 

•  Criteria  for  Measure  Inclusion 

•  Benefits  for  DoD  Program  and  Project  Managers 

•  Assessment 

•  Prediction 

•  Product  Dependability  /  Process  Improvement  Integration 

•  Benefits  for  the  DoD  Acquisition  Manager 

•  New  Measure  Example 

•  Summary 


Introduction 

Standard  Dictionary  of  Measures  of  the  Software 

Aspects  of  Dependability 

Dependability 

-  Trustworthiness  of  a  computer  system  such  that 
reliance  can  be  justifiably  placed  on  the  service 
it  delivers. 

Scope 

-  Specify  and  classify  measures  of  the  software 
aspects  of  dependability. 

-  Provide  conventions  for  the  application  and  use 
of  these  measures. 


Introduction 


Purpose 

Provide  measures  that  are  applicable  for  continual 
self-assessment  and  improvement  of  the  software 
aspects  of  dependability 

The  first  standard  will  be  a  small  document  with 
just  a  few  core  measures  dealing  with  reliability, 
maintainability,  and  availability. 

This  will  be  followed  by  a  second  standard  that 
will  address  safety,  confidentially,  and  integrity. 


Rationale  for  Revision 
Not  revised  since  1988. 

Reaffirmed  in  1996,  but  there  were  significant 
negative  comments. 

Revised  because  many  of  the  original  measures 
had  undesirable  characteristics: 

-  Naivety  about  the  necessary  data,  personnel 
capabilities,  and  training  to  effectively  use  the 
measures. 

-  Did  not  measure  what  they  purported  to 
measure. 

-  Little  field  data  to  back  up  claims  for  benefits. 

-  Measures  were  not  widely  used. 


Criteria  for  Measure  Inclusion 

Have  measures  helped  acquisition  personnel, 
developers,  and  users  achieve  their  reliability 
goals? 

-  Some  minimum  number  of  recognized  uses. 

-  Demonstrated  or  potential  utility  of  the  measure 
in  producing  reliable  software. 

Formulated  generic  measure  classes: 

-  Reliability,  Availability,  and  Maintainability. 


Criteria  for  Measure  Inclusion 

•  Determined  whether  existing  measures  should  be 
modified,  retained,  or  deleted,  based  on  the 
criteria. 

•  Added  any  additional  information  on  the  measures 
developed  since  1988. 

•  Clarified  definition  and  implementation 
conventions. 

•  Identified  and  incorporated,  where  appropriate, 
new  measures  that  have  appeared  since  1988. 
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Benefits  for  DoD  Program  and  Project  Managers 


•  Provide  technology  for  DoD  program  and 
project  managers: 

•  Apply  measures  of  dependability: 

•  To  assess  and  predict  the  dependability  of 
software  during  test  and  operation. 
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Assessment 


Make  an  evaluation  of  dependability  from  a 
historical  perspective  (e.g.,  MTTF). 

For  the  manager,  assessment  answers  the  question: 
how  dependable  has  my  software  been  in  the  past? 

If  for  example,  the  software  has  not  met  reliability 
and  availability  goals,  it  may  be  necessary  to 
strengthen  process  steps,  such  as  inspections  and 
testing. 


Prediction 

•  Forecast  the  future  dependability  of  the  software. 

•  Use  failure  data  obtained  during  test  to  fit  a  model 
for  making  predictions  of  reliability  during 

operation  and  maintenance: 

-  time  to  next  failure,  remaining  failures,  total 
failures  over  the  life  of  the  software 

•  For  the  manager,  prediction  answers  the  questions: 
How  reliable  is  my  software  likely  to  be  in  the 

future? 

•  If  it  will  not  meet  reliability  goals,  what  actions 
are  necessary  to  correct  the  situation? 
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Product  Dependability  /  Process  Improvement 

Integration 

•  For  both  assessment  and  prediction,  there  may  be 
process  improvements  that  could  be  made  to  bring 
the  reliability  of  the  software  up  to  the  goals 
established  in  the  specifications. 

•  Organizations  that  have  the  capability  to  measure 
their  products  and  use  these  measurements  to 
guide  process  improvement  are  those  with  the 
highest  CMM  ratings,  such  as  the  NASA  Space 
Shuttle  avionics  software. 
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Benefits  for  the  DoD  Acquisition  Manager 

•  There  is  much  emphasis  in  DoD  on  integrating 
COTS  into  host  systems  due  to  the  possible 
reduction  in  software  development  cost  compared 
to  developing  the  software  in-house. 

•  These  components  must  be  reliable,  maintainable, 
and  available,  and  must  interoperate  with  the  host 
system  in  order  for  the  customer  to  benefit  from 
the  advertised  advantages  of  lower  development 
and  maintenance  costs. 

•  To  ensure  compliance  with  these  goals,  the 
acquisition  manager  would  specify  in  COTS 
contracts  the  dependability  measures  of  this 
standard. 
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New  Measure  Example 


•  Risk  Factor  Regression  Model 

•  Definitions 

-  CF  =  d*(exp(e*CI)): 

-  Cumulative  requirements  issues  reliability 
prediction  equation. 

-  Issues:  number  of  possible  conflicting 
requirements. 

-  RF  (Risk  Factor):  attributes  of  a  requirements 
change  that  can  induce  reliability  risk. 
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New  Measure  Example 


•  Variables 

-  CF:  Cumulative  Failures. 

-  Cl:  Cumulative  Issues. 

•  Parameters 

•  Coefficients  of  non-linear  regression  equations: 

-  d,  e. 
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Requirements  Change  Threshold 


•  In  F igure  1 ,  cumulative  failures  are  plotted  against 
cumulative  requirements  issues,  for  both  actual 
and  predicted  cases.  When  issues  reach  272,  actual 
cumulative  failures  reach  three  and  climb  rapidly 
thereafter. 

•  In  this  case,  a  cumulative  failure  count  of  three  has 
been  identified  as  a  critical  value. 


15 


Cumulative  Failures 


Reliability  as  a  Function  of  Requirements  Issues 


Figure  1:  Failures  vs.  Issues 


Cumulative  Issues 
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Rate  of  Change  of  Reliability 


•  Because  the  equation  in  Figure  1  is  an  exponential, 
its  derivative  is  also  an  exponential  and  is  simply 
the  original  function  multiplied  by  a  constant.  This 
plot  is  shown  in  Figure  2. 

•  We  should  have  concern  about  the  negative  effect 
on  reliability  of  issues  because  of  its  predicted 
explosive  growth  rate. 
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Cumulative  Failures  per  Cumulativ 
Issues 


Rate  of  Change  of  Reliability  as  a  Function  of 

Requirements  Issues 


Figure  2.  Rate  of  Change  of  Failures  with  Issues 
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New  Measure  Example 


•  Application 

-  Provide  warning  to  software  managers  of 
impending  reliability  problems  early  in  the 
development  cycle  —  during  requirements 
analysis  -  by  using  risk  factors  to  predict 
reliability. 

-  Software  managers  would  be  to  anticipate 
problems  rather  than  react  to  them. 

-  More  efficient  software  management  would  be 
possible  because,  with  advance  warning  of 
reliability  problems,  management  would  be 
able  to  better  schedule  and  prioritize 
development  process  activities. 
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New  Measure  Example 

Data  Requirements 

-  Cumulative  failure  count. 

-  Cumulative  requirements  issues  count. 

Units  of  Measurement 

-  Dimensionless  numbers. 

Experience 

-  Proposed  for  NASA-wide  adoption. 

Tools 

-  SMERFS,  CASRE,  EXCEL,  S-PLUS  6. 


Summary 


•  IEEE  982.1  provides  a  strategy  and  technology  for 
DoD  managers  to  ensure  the  dependability  of  their 
software  during  acquisition,  test,  operations,  and 
maintenance. 
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